Skip to content

ci: Add reruns when running nightly tests if tests fail#2584

Merged
sjrl merged 5 commits into
mainfrom
add-reruns-to-nightly
Dec 9, 2025
Merged

ci: Add reruns when running nightly tests if tests fail#2584
sjrl merged 5 commits into
mainfrom
add-reruns-to-nightly

Conversation

@sjrl
Copy link
Copy Markdown
Contributor

@sjrl sjrl commented Dec 3, 2025 •
edited
Loading

Related Issues

Proposed Changes:

Update all integrations that run unit + integration nightly tests to use cov-retry so we benefit from reruns which help with flaky integration tests.

How did you test it?

Notes for the reviewer

Checklist

@sjrl sjrl requested a review from a team as a code owner December 3, 2025 07:15
@sjrl sjrl requested review from anakin87 and removed request for a team December 3, 2025 07:15
@sjrl sjrl self-assigned this Dec 3, 2025
@anakin87
Copy link
Copy Markdown
Member

anakin87 commented Dec 3, 2025

I'd like not to change all: when running tests locally it's inconvenient to rerun tests multiple times.

We should try to understand if others commands (that we run on the CI) are not currently rerunning tests.

@sjrl
Copy link
Copy Markdown
Contributor Author

sjrl commented Dec 3, 2025

I'd like not to change all: when running tests locally it's inconvenient to rerun tests multiple times.

We should try to understand if others commands (that we run on the CI) are not currently rerunning tests.

Our nightly test in CI uses the hatch run test:all command which is why we don't have reruns currently included. We could reuse cov-retry or make a new one called just retry that we could use for the nightly CI run. WDYT?

@anakin87
Copy link
Copy Markdown
Member

anakin87 commented Dec 3, 2025

Ah you mean "Nightly - run tests with Haystack main branch". Yes, I think we should use cov-retry. My fault.

@sjrl
Copy link
Copy Markdown
Contributor Author

sjrl commented Dec 8, 2025

@anakin87 this is ready for another review!

@sjrl sjrl mentioned this pull request Dec 8, 2025
Merged
anakin87
anakin87 approved these changes Dec 9, 2025
View reviewed changes
Copy link
Copy Markdown
Member

@anakin87 anakin87 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good!

@sjrl sjrl merged commit 44c02a1 into main Dec 9, 2025
33 of 39 checks passed
@sjrl sjrl deleted the add-reruns-to-nightly branch December 9, 2025 08:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment